Visual Comparison of Images Using Multiple Kernel Learning for Ranking

Ranking is the central problem for many applications such as web search, recommendation systems, and visual comparison of images. In this paper, the multiple kernel learning framework is generalized for the learning to rank problem. This approach extends the existing learning to rank algorithms by considering multiple kernel learning and consequently improves their effectiveness. The proposed approach provides the convenience of fusing different features for describing the underlying data. As an application to our approach, the problem of visual image comparison is studied. Several visual features are used for describing the images and multiple kernel learning is adopted to find an optimal feature fusion. Experimental results on three challenging datasets show that our approach outperforms the state-of-the art and is significantly more efficient in runtime. Proposed Approach (RankMKL) Given two images, it is required to learn which image exhibits a particular visual attribute more than the other. Our approach works on a per attribute basis, thus a separate model is learned for each visual attribute. Figure 1 demonstrates the outline of our approach. The first step is to extract a set of features from each image. Several feature sets are selected to capture different visual cues in the image. To capture the image texture, we extract Local Binary Patterns (LBP) [3] and compute the response from a set of Gabor filters. For capturing the shape and appearance of the images, GIST [4] and HoG [1] descriptors are used. Finally, a color histogram is computed in the LAB color space to capture the color information. The second step is to fuse the different feature sets and learn the ranking model. For this task, a separate kernel function is computed for each set of features (i.e. we compute five different kernels). The computed kernels are considered as base kernels for our multiple kernel learning module. Using the multiple kernel learning algorithm, we learn the optimal weights for creating a linear combination from the base kernels together with the optimal parameters for the ranking model. Instead of using a single kernel matrix (K) for learning the ranking model, an optimal combination from several base kernels is learned, and the combination of the base kernels matrix (Kd) is used for training the ranking model, where kd(xi,x j) = φ(xi)d φ(x j)d represents the dot product in feature space φ and is parametrized by d such that: kd(xi,x j) = fd({ki(xi,x j)}i=1), (1) where t is the number of base kernels, d∈Rt is the optimal kernel weights to be learned, and the combination function fd can be a linear or a nonlinear function for combining the base kernels . Our goal is to learn the optimal values for (d) together with the optimal values for the Lagrange multipliers (α) representing the learned ranking model. Accordingly, the standard rankSVM [2] objective function is updated as follows: maximize α {1Tα− 1 2 αT Qdα + r(d)} subject to 0≤ αi, j ≤C,∀(i, j) ∈ P, d≥ 0, (2) Qd,(i, j),(u,v) = kd(xi,xu)+ kd(x j,xv)− kd(xi,xv)− kd(x j,xu), (3) where both the regularizer r and the kernel kd can be any general differentiable functions of d with continuous derivatives and P represents the set of preference paris such that: P = {(i, j)|xi x j}. In our approach, GIST LBP HoG Feature Extraction Which image has a stronger ‘smiling’ attribute?